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Abstract 


Coaching is a promising method for providing professional development, which takes many 
forms. One such fonn is real-time coaching through bug-in-ear technology. This study explored 
the social validity of bug-in-ear coaching when provided as a fonn of professional development 
with pre-service and in-service early childhood educators. Data from two studies were 
qualitatively analyzed to describe early childhood educators’ perceptions of the acceptability of 
bug-in-ear coaching with respect to the learning opportunities provided, feasibility, difficulties, 
and child-level outcomes. Findings suggest that BIE is deemed to be important and effective at 
producing educator and child outcomes. Further, educators are satisfied with the intervention and 
view it to be an acceptable means for receiving professional development. 

Keywords: professional development, personnel preparation, coaching, bug-in-ear, social 
validity, early childhood education 
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The Social Validity of Bug- In-Ear Coaching: Findings from Two Studies Implemented in 

Inclusive Early Childhood Environments 

Professional development is essential for supporting early childhood (EC) educators' use 
of evidence-based practices. Coaching is one promising method for closing the research to 
practice gap by promoting implementation and sustainability of evidence-based practices in the 
classroom (Fixsen, Naoom, Blase, Friedman, &Wallace, 2005; Joyce & Showers, 2002). 
Coaching offers ongoing support rather than one-shot trainings, which are largely ineffective for 
promoting sustained implementation of practices (Odom, 2009). Fixsen et al. (2005) define 
coaching as, “teaching and reinforcing evidence-based skill development and adaptations of 
skills and craft knowledge to fit the personal styles of the practitioners” (p. 47). One important 
feature of effective coaching is immediate feedback provided in real time (Scheeler, Ruhl, & 
McAfee, 2004). Real-time coaching offers the opportunity for immediate feedback to be 
provided to the individual being coached, a practice which increases the effectiveness of the 
professional development as compared to traditional feedback provided at the conclusion of a 
coaching session (Scheeler, et al., 2004). Although there is a substantial body of research on real- 
time coaching (e.g., Scheeler & Lee, 2002; Rock et al., 2009), there is more limited evidence on 
the social validity, or practical importance and social value, of this practice (Horner et al., 2005; 
Wolf, 1978) particularly with EC educators. The purpose of this paper was to explore the 
perceived social validity data from two groups of EC educators teaching in inclusive 
environments who participated in bug-in-ear (BIE) coaching studies. 

Bug-In-Ear Coaching 

BIE coaching is a method for providing immediate feedback to educators within the 
course of teaching (Ottley, in press). BIE methods involve the educator wearing an earbud in one 
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ear, which allows her to hear feedback from the coach in real time while interacting with 
children. Existing evidence indicates BIE coaching is effective in increasing specific teaching 
behaviors in in-service and pre-service educators. For example, Goodman, Brady, Duffy, Scott, 
and Pollard (2008) used a multiple-baseline, single-case design to examine the effects of BIE 
coaching on three pre-service special educators’ use of complete leam units (i.e., a teaching 
sequence that includes an antecedent, student behavior, and consequence). All educators 
increased their use of complete learn units during intervention and two of three maintained these 
improvements when the intervention was faded. Similar results have been observed with in- 
service educators. For example, Scheeler, Congdon, and Stansbery (2010) provided immediate 
feedback on specific teaching behaviors to three co-teaching dyads consisting of one general 
educator and one special educator. All educators increased the percentage of complete learn units 
during intervention and maintained percentages during fading, maintenance, and generalization 
phases. 

More recently, researchers have combined BIE coaching with video technology (e.g., 
Scheeler, McKinnon, & Stout, 2012) to provide immediate feedback from a distance. Coaching 
from a distance takes place when the coach provides feedback in an alternate location from the 
individual being coached. This type of coaching is possible through various web-based forms of 
technology such as video-conferencing software (e.g., Skype™; see Coogle, Rahn, & Ottley, 
2015b). Rock and colleagues (2009) used BIE and Skype™ to provide coaching to 15 pre- 
service educators to enhance their use of specific instructional practices during reading lessons, 
as well as to improve the classroom climate. For each educator, researchers coded videotapes of 
one session with and one session without BIE coaching. Matched-pairs t tests suggested 
significant increases in pre-service educators’ use of the targeted teaching strategies and 
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students’ engagement during intervention sessions. These studies have demonstrated that BIE 
coaching from a distance can be an effective method for supporting educators in their 
classrooms. 

Research examining the use of BIE coaching in EC contexts is more limited in scope. For 
example, programs such as Coaching Approach behavior and Leading by Modeling (CALM; 
Puliafico, Comer, & Albano, 2012) and Teacher-Child Interaction Therapy (TCIT; Gershenson, 
Lyon, & Budd, 2010) use BIE coaching to teach parents and educators to use reinforcement 
strategies to prevent and treat behavior difficulties. Other researchers have used BIE coaching to 
support EC educators’ use of evidence-based communication strategies (Coogle et ah, 2015b; 
Coogle, Ottley, Rahn, & Storie, 2015a). For example, Coogle and colleagues (2015b) examined 
the use of immediate feedback delivered from a distance via BIE and Skype™ on three EC 
special education pre-service educators’ use of environmental arrangement communication 
strategies. The results indicated that all three educators increased their use of communication 
strategies during the intervention phase of research. These and other studies (e.g., Lyon et ah, 
2009) demonstrate that BIE coaching can be an effective means for improving pre-service and 
in-service educators’ (including EC educators) use of evidence-based practices and is effective 
when provided in-person or from a distance using video-conferencing technology. Although BIE 
coaching appears to be effective for increasing a variety of evidence-based practices, in most of 
these studies researchers have focused on specific teaching behaviors and few have examined 
child or student outcomes. Additional research is needed in EC settings to examine the effects of 
BIE coaching on more broad-based implementation of evidence-based practices and resulting 
changes in child outcomes (Coogle et ah, 2015a). In addition, few researchers have examined 
generalization of newly-leamed educator behaviors in other classroom contexts. Social validity 
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data may be one important source of information in helping researchers identify supports needed 
by EC educators to promote implementation and generalization of evidence-based practices. 

Social Validity 

Social validity is defined as the, "practicality, of research procedures and findings" 
(Horner et ah, 2005, p. 172) and has a long history of implementation within the field of applied 
behavior analysis (Baer, Wolf, & Risley, 1968; Gast, 2010; Kazdin, 1977; Wolf, 1978). Wolf 
(1978) has argued that researchers must consider the social importance of their work by 
examining: (a) if the goals of their research are worthwhile to society, (b) if the procedures are 
acceptable, and (c) if consumers are satisfied with the effects. Thus, one means to ensure that 
research is of practical value is to examine its social validity (Horner et ah, 2005; Wolf, 1978). 
There are several approaches to measuring social validity including subjective evaluation, 
nonnative comparison, and sustainability (Kennedy, 2005), which are described in Table 1. 
Social Validity of Bug-In-Ear Coaching 

BIE researchers have primarily used a subjective evaluation approach to determine social 
validity. To obtain social validity data, research teams have asked study participants to either 
complete surveys with Likert-scale and open-ended questions (Scheeler, Bruno, Grubb, & 
Seavey, 2009; Scheeler & Lee, 2002; Scheeler, McKinnon, & Stout, 2012) or to provide written 
reflections in response to question prompts (Rock et ah, 2009). These subjective evaluation data 
are organized based upon educators’ report of BIE’ s effectiveness, acceptability, satisfaction, and 
importance. Researchers have likely used a subjective evaluation approach because of its 
feasibility as compared to other approaches to examining social validity. A normative 
comparison approach requires a comparison group (e.g., educators who did not receive coaching) 
and a sustainability approach requires longer-term measurement of participant outcomes, making 
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these methods more time- and resource-intensive. Thus, it is not surprising that the majority of 
social validity data have been in the fonn of the subjective evaluation approach. 

Pre-service special educators have reported that BIE aided in the acquisition of evidence- 
based practices during coaching sessions, which led to their spontaneous use of practices when 
coaching was not provided (Rock et ah, 2014). Further, social validity data from three BIE 
studies indicate BIE coaching is helpful in increasing student perfonnance with school-age 
students. In one study, educators reported improvements in students' persistence during difficult 
tasks (Scheeler et al., 2010) and in two studies, observations of students’ behaviors indicated 
improved engagement during lessons (Rock et al., 2009, 2014). However, another of Rock and 
colleagues’ (2012) studies found non-significant effects of BIE on student engagement. 

Data on the appropriateness of BIE coaching indicate participants found BIE coaching 
minimally- or non-disruptive (Rock et al., 2009; Scheeler et al., 2009; Scheeler & Lee, 2002). 
School-age children have also reported that they were not distracted by their educators receiving 
BIE coaching (Scheeler, McAfee, Ruhl, & Lee, 2006). In multiple studies, educators have 
reported difficulty with the BIE technology (Rock et al., 2009, 2014), and one educator in 
Scheeler and colleagues' (2009) study reported that she grew tired of using the BIE device. 

Importantly, educators across studies have reported that they enjoyed receiving BIE 
coaching and that they appreciated receiving the feedback. Educators in a multi-year BIE study 
had more positive than negative attitudes toward BIE coaching (Rock et al., 2014). Other 
educators have reported that they would participate in BIE coaching in the future either as a 
recipient of coaching or as a coach to others (Scheeler et al., 2009, 2010, 2012). Further, 
educators in several studies reported an interest in trying BIE coaching with their students or 
with paraprofessionals in their classroom (Scheeler et al., 2009, 2010). 
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Far less information is available on the importance of BIE coaching than on the other 
dimensions of social validity (i.e., effectiveness, acceptability, satisfaction). From the data that 
are available, educators have reported the importance of receiving feedback that is immediate 
(Rock et ah, 2014), thereby highlighting the significance of this key feature of BIE coaching. 

Collectively, social validity data from these studies suggest individuals who participate in 
BIE studies perceive this method of coaching to be beneficial. These data reflect school-age 
educators’ perspectives about the effectiveness and importance of BIE on their own and 
students’ outcomes (Scheeler et ah, 2009, 2010, 2012), as well as educators’ opinions about the 
appropriateness of and their satisfaction with the intervention. In addition, most of these data 
were collected via questionnaires only (c.fi, Rock et al., 2009, 2012, 2014). Unfortunately, little 
data are available on the efficacy of BIE coaching in EC settings, and consequently the social 
validity of BIE for EC educators has yet to be detennined. 

Given the growing body of evidence demonstrating the efficacy of BIE coaching, an 
important next step is increasing the utility and implementation of this practice with EC 
educators. To aid in this process, we chose to thoroughly examine the perceived social validity 
data from two studies with this population to better understand BIE coaching from the 
perspectives of EC educators. Thus, the purpose of this study was to explore the perceived social 
validity of BIE coaching in EC contexts. Our central research question was: How do EC 
educators describe the acceptability of BIE coaching? Sub-questions included: How do EC 
educators describe the learning opportunities experienced through BIE coaching? How do EC 
educators perceive the feasibility of the BIE coaching process? What difficulties do EC 
educators experience with regard to BIE coaching? How do EC educators perceive the 
usefulness of BIE coaching with respect to young children’s communication outcomes? 
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Method 

This research focused on the social validity components of two BIE coaching single-case 
intervention design studies. The purpose of conducting this study was to explore and describe the 
perceptions of EC educators receiving BIE coaching to obtain a deeper understanding of the 
social validity of the intervention. Although the measures of social validity were secondary in 
nature given the scope of the studies, the researchers used five research-based, purposeful 
methods recommended by Brantlinger, Jimenez, Klingner, Pugach, and Richardson (2005) to 
guarantee quality and rigor of data collection and analysis (2005). First, the researchers 
triangulated data in two ways. Methodological triangulation was implemented through the three 
methods including: observations, interviews, and document analysis. Then, data were 
triangulated across data sources (i.e., the participants). Second, the researchers maintained 
detailed notes and records of the data in order to provide thick descriptions of the EC educators’ 
perceived social validity of BIE coaching. Third, the researchers were sensitive to any potential 
disconfirming evidence by thoroughly examining data to determine if inconsistencies from the 
themes emerged. Fourth, the researchers maintained a detailed record of the data (audit trail) to 
demonstrate evidence of data collection and analysis. Finally, data were analyzed in a 
meaningful way (i.e., code-recode strategy). Triangulation, using thick descriptions and 
examining data for disconfirming evidence ensure the data are credible and aid in the 
transferability of research findings. The audit trail, code-recode strategy, data triangulation, and 
disconfirming evidence substantiate that the findings are dependable. 

Participants 

Criterion sampling, a form of purposeful sampling, was used to select the participants in 
order to obtain a deeper understanding of the social validity of BIE coaching for a specific group 
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of participants (Patton, 2002), namely EC educators working in inclusive environments who 
received BIE coaching. Both pre-service and in-service educators were included as participants, 
because previous research has indicated that BIE coaching has been used to provide 
perfonnance-based feedback to both of these populations with similar effectiveness (e.g., 
Scheeler et ah, 2010, 2012) and the research team sought to understand the acceptability and 
satisfaction of BIE from the standpoint of all groups of EC educators receiving BIE coaching. 
Further, the rationale for including participants from inclusive EC settings was two-fold. First, 
this is an area that is just beginning to emerge in the field of BIE research, and second, these 
participants met the criteria of teaching where similar activities were taking place (child-led 
activities, teacher-led activities, routine activities). Thus, these participants had the potential to 
provide rich infonnation related to BIE coaching in EC environments. To the best of the 
researchers’ knowledge, the two studies included were the only BIE studies completed with EC 
educators working in inclusive settings. While the number of participants is small, quality 
indicators in qualitative research do not specify a necessary number of participants. Rather, 
factors such as the purposeful recruitment of participants, adequate representation of the 
population, and saturation are generally used as the guiding principles to detennine the number 
appropriate (Glaser & Strauss, 1967). 

Study 1: BIE Coaching with Pre-service EC Special Educators. The second author 
offered four pre-service educators the opportunity to participate in distance coaching during their 
internship. Three of the four agreed to participate. The pre-service educator who chose not to 
participate was a part of a classroom where a large number of children did not receive consent to 
participate, and she was concerned that it would create challenges in completing her internship, 
as she only would have been able to interact with one child during data collection. Pre-service 
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educators were told that the research team wanted to observe how receiving feedback impacted 
them as a professional. The demographics of the three pre-service EC special educators (Jordan, 
Shay, Noelle; pseudonyms) were similar to those of their peers in relation to their age, race, 
ethnicity, higher-education perfonnance and experience working with young children. All 
educators were non-Hispanic Caucasian, 23-year-old female undergraduate students completing 
a Bachelor’s program in EC Special Education from a large public university in the mid-Atlantic 
region. Educators had an average of 3 years (range 2-6 years) experience working with children 
and an average of 14 months (range 5-24 months) experience working with children birth to 
eight-years-old with disabilities. All of the pre-service educators had received delayed feedback 
from their university supervisors during their three previous practicum placements, but none of 
the pre-service educators had received immediate feedback or BIE coaching prior to this study. 

Study 2: BIE Coaching with In-service EC Educators. The first author offered five 
practicing EC educators working in inclusive settings the opportunity to participate in onsite 
coaching. All five agreed, but prior to the start of the study, one educator quit her job. The four 
remaining educators (Chante, Brandy, Monique, Lanelle; pseudonyms) were informed that the 
purpose of the study was to learn how to enhance the development of young children with 
disabilities or developmental delays. All educators were African-American females who 
averaged 30 years of age (range 22-42 years). Educators had an average of 6 years (range 2-10 
years) of experience working with children birth to five years and they were currently working in 
an Early Head Start (n = 3) or a public EC center (n = 1) in the Southeast. The highest levels of 
education completed were a Child Development Credential (n =1), Associate’s degree (n = 2), 
and Bachelor’s degree (n = 1). The educators had never received BIE coaching prior to this 
study; however, their previous experiences receiving other forms of coaching were unknown. 
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Intervention 

The professional development provided to the pre-service and in-service EC educators 
aligned with the Division for Early Childhood’s recommended practices for teaming and 
collaboration (2014) and their position statement on cultural and linguistic responsiveness 
(2010). Specifically, the researchers communicated regularly with EC educators to enhance the 
trust in their relationships and demonstrate respect for the educators. Additionally, 
encouragement and performance-based support were provided regularly to promote educators’ 
confidence and capacity to implement the communication strategies. 

For Study 1, pre-service educators learned the communication strategies through a 
narrated PowerPoint presentation that was provided to them after completion of the baseline 
phase. This form of instruction was similar to the types of instruction received via their 
university courses. BIE coaching for Study 1 occurred at a distance with the second or third 
author providing feedback to pre-service educators from an off-school location via Skype™. 
Coaching sessions were 10 min in length and lasted one week with coaching sessions occurring 
twice a day for four days. 

The intervention for Study 2 included three 15 -min face-to-face trainings that focused on 
one instructional strategy each. BIE coaching occurred for an average of 15 min a day for three 
to five days per week. The coaching duration was approximately two months in length. 

Data Collection 

Triangulation of data was conducted by using various methods of data collection in each 
study as well as including data from multiple participants (Miles & Huberman, 1994; Patton, 
2002; see Tables 2 and 3). Data collection in Study 1 included observations, interviews, and 
document analyses; whereas, Study 2 used observations and document analyses. Interviews and 
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questionnaires from Study 1 were conducted upon completion of pre-service educators’ 
internship experiences and grade assignments in order to diminish the possibility of pre-service 
educators providing false-positive responses if it were believed that positive responses could 
influence their final internship grades. 

Observations. Observations were conducted by watching 10-min (Study 1) or 3-min 
(Study 2) videos of the educators during BIE coaching. Data were collected until saturation was 
reached with each participant (i.e., until the research questions were answered; Charmaz, 2006; 
Glaser & Strauss, 1967). Because two university faculty provided feedback for Study 1 
(observational data), the researchers watched at least one video from each researcher. Study 2 
included three waves of intervention per educator (one for each communication strategy). One 
video was observed from each wave for all educators. 

Interviews. Interviews are frequently used in qualitative research to allow participants 
the opportunity to elaborate on information related to the research questions (Patton, 2002). 
Therefore, semi-structured, open-ended interviews were conducted (Study 1). Questions focused 
on pre-service educators’ experiences with BIE coaching. Each educator was provided the 
questions prior to the interview and were interviewed face-to-face to obtain more direct 
information and develop clarity on observation notes. Interviews lasted 45 to 60 minutes. The 
second author transcribed all responses to prepare for analysis. 

Document analysis. Document analysis is often used in qualitative research to 
triangulate data (Patton, 2002). Using the methods for analysis (see data analysis), the 
researchers coded the open-ended questions from the social validity questionnaire that each 
educator completed. Document analysis was conducted by reviewing one questionnaire 
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completed by each educator. Additionally, co-educators from Study 2 completed social validity 
questionnaires describing their perspectives of the BIE coaching. 

Data Analysis 

All data were used to answer the research questions. Interviews, observations, and 
responses to the open-ended questions from the social validity questionnaire were transcribed 
into Microsoft Excel and were coded and recoded according to the iterations identified in Table 
4. The second author coded data into starter codes based on the research questions (Open 
Coding: Iteration One). The four starter codes were learning, feasibility, challenges, and child 
outcomes. The first author reviewed a random 33% of the initial analysis to ensure inter-rater 
reliability (Iteration Two), which was achieved at 95% agreement. Disagreements were 
discussed until agreement was reached regarding the most suitable code for the data (Iteration 
Three). Using the starter codes as themes, the first author identified subthemes by using pattern 
coding (Iteration Four). Subthemes related to each theme are identified in Table 5. These codes 
were then analyzed for reliability by the first author (Iteration Five) with reliability greater than 
or equal to 90% for all codes. Last, the first and second authors discussed and recoded any data 
where disagreements occurred (Iteration Six). 

Findings 

The findings from this study provide a deeper understanding of how EC educators 
describe the acceptability of BIE coaching. Findings are organized by the themes and emerging 


subthemes (see Table 5). 
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Theme One: EC Educators’ Learning Experiences with BIE Coaching 

Data from the theme of learning experiences included the two sub themes of mechanisms 
for providing a learning opportunity (52% of the codes in the theme) and enhanced abilities 
(48% of the codes). Data from both subthemes were documented from all seven EC educators. 

Mechanisms for providing a learning opportunity. Data within mechanisms for 
providing learning opportunities included the following units of meaning: prompting feedback (n 
= 46), positive feedback (n = 50), corrective feedback ( n = 7), practice applications (n = 1 1), and 
problem-solving (n = 8). Prompting feedback was exhibited when Brandy was prompted to 
“model bread” during socio-dramatic play, positive feedback was illustrated when coaches 
provided educators with praise for using strategies effectively, and corrective feedback was 
illustrated when Jordan was provided the following feedback, “When you offer him a choice this 
time, name the two bugs so he will use one of the bug names.” A practice application was 
demonstrated when Chante indicated that the immediate feedback “reminded [her] to use the 
strategy.” Problem-solving contextual situations during BIE sessions were documented when 
educators would ask questions or coaches would provide extra support as a means to clarify the 
situation. This included coaches providing examples of the strategies, such as suggesting that the 
educator offer the choice of “burying an animal or a person” during sensory table play. 

Enhanced abilities to use strategies, effective teaching practices, and follow 
feedback. Enhanced abilities was represented through educators’ response to prompts (n = 10), 
accurate use of communication strategies (n = 41), spontaneous use of strategies (n = 45), 
improved teaching practices ( n = 9), and confidence (n = 8). When coaches would prompt the 
educators to use a strategy, they would use it effectively. For example, in one of Andrea’s 
observations, she said, “The oven is hot; can you say hot?” Educators were observed 
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spontaneously perfonning the strategies without prompting (e.g., Lanelle’s use of imitation 
during a read aloud). Educators also described in interviews and questionnaires their independent 
use of the strategies when coaching was not occurring. Shay reflected in her questionnaire that 
she was starting to use the strategies “more naturally, without even thinking about it and in 
situations such as meal time and outside play.” 

In addition to improving educators’ abilities to use the strategies, educators described 
enhanced overall teaching abilities. For example, Shay perceived improvements in her lesson 
planning and Brandy perceived herself as better able to provide accommodations for children 
with disabilities. Additionally, educators expressed confidence in their abilities to use the 
communication strategies and meet the communication needs of children with developmental 
delays and disabilities. Noelle was somewhat confident in her abilities, communicating in her 
questionnaire, “Once I practice these [strategies] more, I feel I will be more confident.” 

Theme Two: EC Educators’ Perceptions of the Feasibility of BIE Coaching 

Three subthemes emerged from the data describing the feasibility of BIE coaching. All 
EC educators indicated they were able to continue with their typical routines and procedures 
(77% of the data in this theme). Six of the seven educators expressed appreciation for receiving 
immediate feedback (13% of data) and five educators commented on the influence of BIE on 
young children (10%). 

Ability to maintain classroom routines and procedures. Data emerged which revealed 
EC educators receiving BIE coaching were able to maintain their scheduled academic activities 
(n = 18) and classroom behavioral plan (n = 6). Further, educators continued questioning 
children (n = 9), maintaining consistent procedures for activities (n = 3), and inviting children to 
participate in the activities ( n = 2). All of these codes were from observation data of the 
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educators and children during coaching sessions. For instance, while singing songs with the 
children, Lanelle was able to receive feedback and embed the communication strategies into the 
activity. The remaining codes in this subtheme (n = 10) were general comments made about the 
feasibility of BIE coaching. For example, Monique stated, “[BIE] allowed me to get instruction 
and feedback without disturbing the children during our activities.” Additionally, Jordan stated, 
“I think it worked out a lot better than what I anticipated. It was a lot less invasive [emphasis 
added] than what I imagined.” 

Appreciation for the immediate feedback. The eight codes in which EC educators 
demonstrated appreciation for receiving BIE coaching were related to the benefit of receiving 
immediate feedback. Noelle stated, “from immediate feedback you get to try right then and there 
what they are suggesting instead of hearing about it later and trying to replay a situation where 
the feedback can be used.” Additionally, a co-educator in Chante’s classroom stated that she 
would be interested in BIE coaching “. . .for personal improvement, to see the difference it makes 
with the kids;” a comment appearing to suggest that she observed improvements in the children 
with whom Chante interacted. 

Influence on children. Jordan and Noelle indicated that the children were interested in 
the iPad and looking into the camera, but that BIE coaching did not affect the children or what 
they would have typically been doing in the classroom. Observational data supported these 
findings, with the children in Monique, Lanelle, and Brandy’s classrooms engaged in typical 
play-based routines, without distraction toward the camera or the coach. 

Theme Three: EC Educators’ Difficulties with BIE Coaching 

Data from this theme represent four of the EC educators’ perspectives on BIE challenges 
as communicated through interviews, questionnaires, and observations. Specifically, three 
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educators experienced challenges with the technology (n = 15), multitasking (n = 16), and the 
technology distracting the children (n = 7). Additionally, two educators had challenges obtaining 
child consent (n = 9) and keeping-up with data collection (n = 2). 

Technology challenges. The most common technology difficulty suggested by 
participants was problems hearing the coach (n = 6). Additionally, educators experienced 
difficulties related to the internet (n = 3), such as it not working and sometimes when it was 
working, Skype™ calls were dropped. Further, during some activities, the iPad camera was 
unable to capture a full view of the contextual environment (n = 3). Finally, Chante voiced 
through her interview that she both enjoyed and failed to enjoy the BIE coaching. Specifically, 
she did not enjoy BIE coaching when the Bluetooth™ “echoed at noisy times.” 

Educators’ challenges multitasking. At times, educators indicated having difficulties 
getting children to participate (n = 2) and keeping the pace of a lesson when using wait time (n = 
1). On one occasion, a coach from Study 1 struggled to provide feedback, because of the type of 
activity in which the educator and child were engaged. By far, the most challenging aspect of 
BIE coaching, suggested by the number of codes within this unit of meaning, was related to the 
challenges associated with leading classroom activities and listening to/embedding feedback 
simultaneously (n =12). Observation data suggested that during the hectic times in the 
classroom, the educators appeared to implement some of the coach’s feedback suggestions, while 
ignoring others. This was reiterated in Noelle’s questionnaire. 

The challenge is that sometimes the classroom is a little hectic so when you are the lead 
teacher and trying to listen to what the person is saying to you as well as what is going on 
in the classroom, it [.fie] can be difficult at times. 
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Technology distracting children. According to the participants in Study 1 and Study 2, 
children in BIE classrooms were interested in, and at times distracted by (n = 5), the technology. 
Behaviors exhibited by children included: looking at themselves in the camera, playing with the 
iPad, and moving the angle of the iPad. Jordan thought that children with developmental delays 
or disabilities were less aware of the technology. Further, Chante thought that the children’s 
distractibility and challenging behaviors were more notable at first, but subsided throughout the 
intervention. 

Obtaining familial consent for children to participate. Noelle communicated that the 
study was somewhat difficult to implement because not all children in the classroom provided 
consent (n = 5). Furthermore, during Noelle’s interview, she reported that at times she had to 
wait to complete the BIE coaching until children with consent were available to participate in the 
activity, which made the coaching “more stressful.” 

Scheduling difficulties. Two codes emerged from the data to form the sub theme of 
scheduling and both codes describe the intensity of the intervention. Shay and Noelle thought 
that the baseline data collection and BIE coaching occurred at a rate that was difficult to manage. 
Theme Four: Educators’ Perceptions of the Usefulness of BIE Coaching for Improving 
Children’s Development 

A total of 62 codes emerged within the perceived usefulness of BIE coaching for 
improving children’s development. Within this theme, all EC educators identified improved 
communication ( n = 40) and engagement (n = 19) as child-level outcomes of BIE coaching. 
Additionally, three educators indicated that children learned “other skills” (n = 3). 

Children’s communicative development. All of the codes representing educators’ 
perceptions of BIE on children’s communicative development (n = 40) described children’s 
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enhanced verbalizations, as well as the impact of strategies on promoting children’s 
communication. Brandy’s co-educator reflected in her questionnaire, “They picked a choice ... 
They either pointed to it, or if they were able, they said it.” Furthermore, Chante indicated the 
impact of expansions when she said it, “gave them more words to use instead of one or two 
words.” These data were supported with observations, such as a child in Monique’s class saying, 
“my rocket is going up [into] space. . .up, up, up” during a play-based activity as well as a child 
in Lanelle’s class shouting out the number of monkeys left in the bed during a read aloud. 

Children’s engagement in activities. Educators described children’s engagement in the 
following ways: attending to the lesson (n = 3), staying the entire lesson (n = 2), waiting their 
turn (n = 1), actively participating (n = 3), listening to instruction (n = 1), and cleaning up (n = 

1). For example, children in Monique’s classroom were actively participating in the 
housekeeping center during socio-dramatic play, engaging with their peers and educators. 
Additionally, Jordan described a child’s increased engagement in the following example: 

The child that I was working with in blocks normally does not stay in blocks for more 
than a minute or two without becoming very frustrated. By using choice making with 
every block I gave him, he stayed engaged and we played in blocks for 10 min. This was 
huge for him! 

Children learning other skills. Educators reported that children acted more politely 
during small-group activities (Noelle) and learned content-area vocabulary through the repetition 
involved in offering choices (Jordan). Moreover, Shay shared, “This was a short study but if 
these strategies are used daily the overall results for the children could be drastic.” 
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Discussion 

The purpose of this study was to explore and describe the perceived social validity of BIE 
coaching with EC educators. Specifically, we sought to gain a deeper understanding regarding 
the acceptability of BIE coaching with respect to the learning opportunities provided, feasibility 
in EC contexts, difficulties encountered, and usefulness in supporting young children’s 
development. Findings are summarized by the social validity dimensions of perceived 
effectiveness, acceptability, satisfaction, and importance (Homer et ah, 2005). 

Perceived Effectiveness of BIE Coaching 

In these studies, the EC educators self-reported that BIE coaching was effective in 
promoting their own instructional abilities as well as children’s development. Through BIE 
coaching, educators appeared to leam how to apply the strategies within their classroom 
activities and they seemed to internalize the strategies, which was evidenced by their 
spontaneous use of them during both coached and non-coached sessions. Further, most educators 
self-reported an improvement in their confidence to use the communication strategies and/or to 
meet the communicative needs of children with disabilities. This finding is important given that 
educators who report greater confidence tend to provide higher-quality instruction (Holzberger, 
Philipp, & Kunter, 2013). Moreover, these findings support the research literature suggesting 
BIE coaching to be an effective means for enhancing educators’ instructional practices (e.g., 
Rock et ah, 2009). 

Similar to the school-age BIE literature (e.g., Rock et al., 2009), the EC educators who 
received BIE coaching perceived the children to have improved outcomes in the area that was 
targeted for intervention. This finding aligns with correlational research demonstrating positive 
associations between BIE coaching and young children’s communicative development (Ottley 
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Ferron, & Hanline, in press; Ottley & Hanline, 2014). Extending the literature, our research also 
indicates that educators thought the children had improved outcomes in non-targeted skills, such 
as vocabulary and social development. For example, Noelle indicated that the children were 
“more polite.” Thus, EC educators’ perceptions suggest that children may improve in other 
skills, such as social skills, that were not specifically targeted, but that are related to the 
instructional strategies implemented by EC educators. 

Data on the feasibility of BIE coaching in EC environments seem to suggest that although 
the children were interested in viewing themselves in the camera, the technology did not serve as 
a barrier to children’s participation in scheduled activities. Additionally, we had triangulated 
evidence that EC educators perceived children to be engaged with materials and activities in 
meaningful ways during BIE coaching sessions. This outcome supports that of Rock and 
colleagues (2009, 2014), who found children to be significantly more engaged during lessons 
when BIE coaching was implemented. Collectively, these findings either support or extend the 
literature base, with all data appearing to indicate that in promoting educators’ use of evidence- 
based practices via BIE coaching, child outcomes may also be enhanced. Thus, EC educators’ 
perceptions that BIE coaching positively influence children’s development provides important 
information regarding BIE’s social validity for EC educators who work in inclusive 
environments. 

Perceived Acceptability of BIE Coaching 

As EC educators were participating in BIE coaching, they reported that they were able to 
maintain their typical classroom routines and procedures. We perceived this to be a sign that BIE 
coaching could feasibly be embedded into EC environments. Consistent with the school-age 
literature (e.g., Scheeler et ah, 2009), educators did not find the intervention intrusive and 
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reported that it was only minimally distracting to the children. The EC educators’ statements 
seemed to indicate that the real-time coaching was less invasive than they thought it would be 
and that the BIE technology provided privacy, which enabled normal routines to continue in the 
classroom while coaching was provided. 

The findings from this research provide a deeper understanding regarding challenges with 
BIE coaching that had yet to be described within the BIE coaching literature. Specific challenges 
noted were obtaining consent, the high frequency of scheduled coaching sessions, and multi- 
tasking. These challenges present some important information with respect to delivering BIE 
coaching in EC contexts. First, it may be helpful to conduct BIE in EC classrooms where the 
majority of children have consent to participate. This may minimize the challenges involved with 
only including certain children in the activities in which video-recording occurs. Second, two of 
the seven educators struggled with the pacing of data collection sessions. Thus, two coaching 
sessions per day (which was used in Study 1) may be too frequent for EC educators to manage. 

Contrary to the EC educators in our study who found it difficult to receive BIE feedback 
when the classroom was chaotic, the pre-service educators in Rock and colleagues’ (2014) study 
found BIE helpful in chaotic lessons. These opposing data may be the result of differences in the 
strategies coached or in the educational contexts (school-age versus EC). Importantly, the EC 
educators found a way to manage these chaotic situations by responding to the coach’s feedback 
when possible, but ignoring some feedback when necessary because of the classroom demands. 
This finding highlights the flexibility and autonomy provided with BIE coaching in that 
educators can choose whether or not to respond to the feedback at their own discretion 
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Perceived Satisfaction with BIE Coaching 

The richest information regarding EC educators’ satisfaction with BIE coaching was their 
report that they would use BIE again and that they would recommend it to their colleagues and 
the children’s families. This finding aligns with other research, suggesting that educators who 
receive BIE (e.g., Rock et ah, 2012; Scheeler et ah, 2010) and other forms of coaching (e.g., 
Diamond & Powell, 2011; Marturana & Woods, 2012) enjoy the coaching and/or would 
recommend it to others. Supporting the school-age BIE literature (e.g., Rock et ah, 2014), when 
coaching is provided from a distance, there appears to be more challenges than when BIE 
coaching is provided onsite. For example, distance coaching requires reliable internet and the use 
of additional technology (e.g., web-camera) and a video-conferencing platform. Nonetheless, the 
educators in this study who received coaching from a distance still reported their appreciation for 
the coaching, their enjoyment receiving feedback, and their acknowledgement of willingness to 
use BIE again to enhance other instructional practices. 

Perceived Importance of BIE Coaching 

With respect to the process of coaching, EC educators reported the benefit of receiving 
immediate feedback. Immediate feedback supported their learning through the real-time practice 
opportunities in the classroom context. Jordan had previously received delayed feedback, which 
was received after her university supervisor had observed her classroom practices. She indicated 
that the immediate feedback was “more effective” than the delayed, because with delayed 
feedback, “it is hard to go back and think about what the exact situation was and all the details as 
to what was happening with the children.” This finding supports the importance finding 
documented with the school-age population (Rock et al., 2014). 
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In addition to the immediate feedback educators receive, the other main component of 
BIE coaching is the use of technology to provide the feedback. These BIE technologies allow for 
privacy and discretion in the receipt of feedback, which is made possible through the earbud. 
Although six of the seven EC educators reported the importance of receiving immediate 
feedback (see above), only two educators reported the importance of receiving feedback without 
disrupting the classroom activities. The lack of comments on the privacy of BIE coaching may 
have resulted from these educators not having experienced immediate coaching without the 
earbud and thus being unaware of the benefits it provides. Nonetheless, the two EC educators 
who thought receiving feedback via the earbud was important indicated that this feature was 
critical so that the children and other educators in the environment were not disrupted by the 
coaching. These educators also reported that the BIE methods provided them with privacy, so 
that they could receive coaching without their co-educators hearing what was being suggested 
for them. Privacy was extremely important for Monique who disliked her co-educator and did 
not want the co-educator to know her conversations with the coach. Although other types of 
coaching used in EC environments (e.g., use of hypermedia with video feedback; Diamond & 
Powell, 2011) offer the opportunity for feedback to be provided in a discrete manner to educators 
in either written or verbal fonnat, these methods remove the immediacy in which feedback is 
received. This is problematic, because of all types of performance-based feedback, immediate 
feedback is the component with the most empirical evidence of effectiveness (Scheeler et ah, 
2004). Importantly, these findings seem to suggest that the non-disruptive and private nature of 
BIE coaching may be particularly appropriate for EC classrooms where multiple educators 
typically teach in the same classroom setting. 



BIE SOCIAL VALIDITY 


24 


The third aspect of BIE coaching that educators reported to be important was the 
outcomes observed on children’s development. At first, several educators were surprised that the 
children actually communicated in response to their use of strategies. However, as children 
responded to the educators’ use of the communication strategies, the educators seemed to value 
the strategies more and began to spontaneously use them when coaching was not provided. 
Furthermore, a co-educator who did not receive BIE coaching, but who observed the coaching 
sessions, indicated that she would be interested in BIE to see the effect on the children. This 
demonstrates that the perceived effectiveness of the strategies on children’s development may be 
crucial in getting educators to “buy-in” to evidence-based practices and the BIE mode of 
coaching. 

Conclusion 

Collectively, these findings appear to suggest that EC educators perceive BIE coaching to 
be important and effective in enhancing both their development as educators, and children’s 
targeted outcomes. Educators report satisfaction with BIE as well as acceptability of the 
intervention. These data align with Homer and colleagues’ (2005) recommendations for 
measuring a practice’s social validity and indicate that BIE coaching is a socially valid method 
of providing professional development for EC educators. 
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Table 1 


Approaches to Social Validity 


Approach 

Description 

How Social Validity is 
Determined 

Subjective evaluation 
(Kazdin, 1977; 

Kennedy, 1992; Wolf, 
1978) 

Researcher collects data on 
individuals' (e.g., experts, 
educators, parents, participants) 
perceptions of the intervention 

Considered socially valid if 
individuals perceive the 
intervention to be acceptable and 
they are satisfied with outcomes 

Normative Comparison 
(Kennedy, 1992; Van 
Houten, 1979) 

Researcher compares 
participants' behaviors with 
those of a typical comparison 
group 

Considered socially valid if 
participants' behaviors after 
intervention are within the 
typical range of individuals not 
in need of intervention 

Sustainability 

Approach (Kennedy, 
2005) 

Researcher examines outcomes 
after the intervention to 
determine whether outcomes are 
maintained following treatment 

Considered socially valid if 
outcomes are sustained 



Table 2 


Triangulation of Data across Methods 


Research 

Questions 

Study 1: 
Interview 
Questions 

Study 1: 
Video 

Observations 

Study 1: 
Document 

Analyses 

Study 2: 
Document 

Analyses 

Study 2: 
Video 

Observations 

Learning 

Experiences 

X 

X 

X 

X 

X 

Perceived 

Feasibility 

X 

X 

X 

X 

X 

Difficulties 

Experienced 

X 

X 

X 

X 


Usefulness for 

Children’s 

Development 

X 

X 

X 

X 

X 



Table 3 


Triangulation of Data across Participants 


Research Questions 

Emily 

Megan 

Kara 

Chante 

Brandy 

Monique 

Lanelle 

Learning Experiences 

X 

X 

X 

X 

X 

X 

X 

Perceived Feasibility 

X 

X 

X 

X 

X 

X 

X 

Difficulties 

Experienced 

X 

X 

X 

X 




Usefulness for 
Children’s 

Development 

X 

X 

X 

X 

X 

X 

X 



Table 4 


Data Iterations and Procedures 



ITERATIONS 

PROCEDURES 

One 

Open Coding 

Author 2 organized data into the four starter codes, which served as the themes of the 
study. 

Two 

Inter-rater Reliability 

Author 1 checked 33% of Author 2’s initial analysis for inter-rater reliability. The authors 
reached high agreement of 95%. 

Three 

Resolving Disagreements 

For disagreements, the authors discussed rationales behind theme coding until an 
agreement was reached. 

Four 

Pattern Coding 

Author 2 examined data within each theme to identify subthemes using pattern coding. 

Five 

Inter-rater Reliability 

Author 1 checked 33% of Author 2’s initial analysis for inter-rater reliability. The 
authors reached high agreement of 94% in learning; 90% in feasibility; 92% in 
challenges; and 100% in child outcomes. 

Six 

Resolving Disagreements 

For disagreements, the authors discussed rationales behind subtheme coding until an 
agreement was reached. 



Table 5 


Amount of codes within each theme and subtheme 


THEMES 

N (% all data) 

Sources of data 

SUBTHEMES 

N (% of theme data in the subtheme) 
Sources of data 




Educators’ Learning 
Experiences 
n = 235 

57.6% 

Learning 

Mechanisms 
n = 122 

51.9% 

Enhanced 

Abilities 
n = 113 

48.1% 




Educators’ Perceptions of 
BIE Feasibility 
n = 62 

15.2% 

Maintain schedules 
and procedures 
n = 48 

77.4% 

Appreciation for 
immediate feedback 
n = 8 

12.9% 

Influence on the 
children 

n = 6 

9.7% 



Educators’ Difficulties 
with BIE Coaching 
n = 49 

12.0% 

Technology 
Challenges 
n = 15 

30.6% 

Multitasking 
Challenges 
n = 16 

32.7% 

Technology 
Distractions 
n = 7 

14.3% 

Obtaining 
Consent 
n = 9 

18.4% 

Scheduling 
Challenges 
n = 2 

4.1% 

Educators’ Perceptions of 
Usefulness with Children 
n = 62 

15.2% 

Increased 
Communication 
n = 40 

64.5% 

Increased 

Engagement 
n = 19 

30.6% 

Learning 

Other Skills 
n = 3 

4.8% 





